CONSTRAINED AND UNCONSTRAINED PDE’S FOR VECTOR IMAGE RESTORATION D.Tschumperlé and R.Deriche

The restoration of noisy and blurred scalar images has been widely studied, and many algorithms based on variational or stochastic formulations have tried to solve this ill-posed problem [2, 4, 10, 7, 33, 20, 19, 18, 22, 1, 26, 24, 9, 28, 29, 6, 30, 35, 37]. However, only few methods exist for multichannel/color images ([7, 29, 16, 36]). Here, we propose a new vector image restoration PDE which removes the noise and enhances blurred vector contours, thanks to a vector generalisation of scalar -function diffusions and shock filters. A local and geometric approach is proposed, which uses pertinent vector informations. Finally, we extend this equation to constrained norm evolutions, in order to restore direction fields and chromaticity noise on color images. 1. PRINCIPLE OF ANISOTROPIC DIFFUSION We consider a scalar image I(M) : ! R ( 2 R2 ). Scalar image restoration using -functions classically consists in minimising the following functional : min I Z 2 (I I0)2 + (krIk) d where : R ! R is a regularisation function that penalizes high gradients, while preserving edges. The minimisation can be performed via the corresponding anisotropic PDE evolution, coming from the Euler-Lagrange equations : I t = (I0 I) + div 0(krIk) krIk rI! which can be also written as : I t = 0(krIk) krIk I + 00(krIk) I + (I0 I) where I = 2I 2 = r(rI: ): with = rI krIk and = ?. In [19], this expression was interpreted as two directional 1D heat flows with different diffusion intensities : = 00(krIk) and = 0(krIk) krIk in the corresponding directions and . A diffusion function must verify these natural properties : When krIk ' 0, the local geometry is flat and doesn’t contain any edges, the diffusion must be isotropic : ' ' 1 =) I t ' I + I = I When krIk 0, the current point may be located on an edge, the diffusion must be anisotropic (oriented by the edge) : =) I t ' I Many -functions were proposed in the literature : Total variation [27], Perona & Malik [24], Geman & McClure [31], Green [13], Hebert-Leahy [14],: : : In [19], the authors also proposed to fix directly the smoothing intensities : = g (krIk) (decreasing function) and = 1. It ensures a permanent noise removal, but tends to smooth sharp corners. Geometrically speaking, a PDE restoration process must adapt its diffusion behaviour to the local geometry of the image. For the scalar case, this geometry is given by an edge indicator N(I) = krIk, and the associated directions and , respectively orthogonal and parallel to the edges. A vector image diffusion process needs to define such equivalent vector attributes : a vector gradient norm N(I), and the corresponding smoothing directions ; for the whole image components, taking the coupling into account. Using a channel by channel approach is then useless : each channel of the image evolves with different smoothing directions and intensities. The diffusion is not coherent with a vector geometry and edges tend to be smoothed (Fig.1). a) noisy image b) decoupled PDEs c) coupled PDEs Fig. 1. Channel by channel approach vs coupled PDEs, on a noisy color image (in the RGB space). This paper is organised as follow : We first show how to define a local vector geometry, using the classic Di Zenzo method [38], then we compare and interpret some recent vector diffusion PDEs. This comparison yields a new geometric and intuitive vector restoration PDE (eq.(7)) We finally extend this idea to norm constrained evolutions, and propose some results. 2. DEFINING A VECTOR GEOMETRY Now, we are interested in vector images I(M) : R2 ! Rn . I i denotes the ith image channel (1 i n). We want to define a vector gradient norm N(I) and variation directions and , corresponding to a local vector geometry. 2.1. First approach : scalar conversion The first idea is to find a function f : Rn ! R so that the image f(I) is representative of the human perception of vector edges (for instance, f = L , the luminance, for color images). Then can be chosen to be the direction ofrf(I), and N(I) = krf(I)k. The choice of such functions is not an easy task ! However, there are mathematically no functions detecting all possible vector variations. For the color example, it wouldn’t be able to detect iso-luminance contours. 2.2. Differential geometry of surfaces Di Zenzo [38] considers a vector image I as a 2D ! 3D surface, and looks for the local variations of kdIk2 : kdIk2 = dx1 dx2 T g11 g12 g12 g22 dx1 dx2 with gij = I xi : I xj The two eigenvalues of (gij) are the extremum of kdIk2 and the orthogonal eigenvectors , are the corresponding variation directions : 8><>: += = g11+g22 p(g11 g22)2+4 g2 12 2 = 12 ar tan 2 g12 g11 g22 = + 2 (1) Then, several vector gradient norms N(I) can be defined : In [29], the authors uses a decreasing function f( + ) to weight their diffusion PDE. It can be seen as a function of a vector variation norm N(I) = p + . Note that this norm fails to detect corners where + = (see the checkboard intersections in Fig.2). In [32] and [7], the norm N(I) =p + + is proposed for a global minimisation process, but can be also used as a local norm definition. It is very easy to compute, since N(I)2 = + + = n Xk=1 krIkk2 Note that this norm gives more importance to certain corners (but not all) (Fig.2). We propose to use N(I) = p +, as a direct extension of the gradient norm definition : the value of maximum variation. It doesn’t give more or less importance to corners. Color image p + p + p + + Fig. 2. Differences between vector variation norms It is worth to mention the work of Kimmel-Malladi-etal [16], which consider a n-D vector image as a surface embedded in a n+2 dimension space. They introduce the induced metric (g i;j) in a Polyakov functional minimisation, in order to construct a scale space of the vector images. This metric is directly linked to the Di Zenzo approach : g i;j = Æi;j + gi;j where Æi;j = 0 if i 6= j 1 if i = j One may note that the corresponding eigenvalues and eigenvector directions are given by : + = + + = 1 + + and = = 1 + These expressions show the similarity between the two approaches. The Di Zenzo equations define then a pertinent vector geometry with a variation norm N(I) and corresponding directions and , which can be used in the restoration process to take the coupling between image channels into account. Color edge detection is a direct application of the vector gradient norm definition : One just has to look for the local maxima of N(I) in the direction (fig.3). Original color image Color edges Fig. 3. Color edge detector : Thresholded local maxima of p + in the direction 3. DIFFUSION EQUATIONS We analyse now some proposed vector diffusion equations ([29, 7]), in order to introduce our approach and propose an original and efficient vector diffusion PDE. Comparisons results on synthetic images are shown in the end of this section. 3.1. Sapiro-Ringach’s vector diffusion PDE In [29], the authors propose this anisotropic vector diffusion PDE : I t = g( + ) I (2) where g(:) is a positive decreasing function with lim s!+1 g(s) = 0 and I = 2I 2 ( is found with the Di Zenzo calculus). It was a first step in viewing the importance of coupling in a diffusion process. The diffusion factor g( + ) and the smoothing direction contain informations of coupling between vector components. A vector geometry is taken into account : At a given point, all channels evolve in the same direction and with the same intensity. Edges are then not smoothed (but are not perfectly detected, as described in section 2.2). Anyway, few problems remain : Along very high gradient edges (N(I) 0), smoothing may be weak and doesn’t remove the noise : I t ' ~0 (the choice of a function g which doesn’t decrease too fast is primordial here). In homogenous regions (N(I) ' 0), the image pixels diffuses only in the direction , which is very sensitive to the noise when the geometry is flat ! : I t ' I . Undesirable texture effects may appear in these regions, because of the uni-directional diffusion. No data attachment term : the PDE evolution must be stopped before convergence for a good result. 3.2. Blomgren’s TVnm diffusion equation As defined in [7], the TVn;m diffusion PDE with a component by component writing style is : I i t = TVn;1(I i) TVn;m(I) div rI i krI ik + (I i 0 I i) (3) with TVn;m(I) =vuut m Xk=1 Z krIkk 2: This PDE comes from a minimisation process, which use coupling between vector components in the functional expression. But, if we introduce the i direction ( i? i = rIi krIik ), then I i t = (I i 0 I i) + Ai krI ik I i i Ai = TVn;1(I i) TVn;m(I) The only coupling terms in the final PDE isAi, which weight the diffusion intensity in each image channel. The diffusion is uni-directional and the smoothing direction is independent for each channel, which leads to the problem of decoupled diffusion (Fig.1). Despite the uni-directional diffusion, texture effects are less visible in flat regions, because each channel diffuses in a different direction i. For color images, it corresponds to a color blending effect. Anyway this advantage becomes a drawback in contour regions : A local vector geometry is not taken into account, and edges evolve individually in different directions, component by component. Edges tend to be smoothed. 3.3. A geometric diffusion PDE approach Here is our approach, considered as an extension of our previous work [34, 20, 19, 18]. It is based on a geometric viewpoint of the diffusion process. The vector gradient norm N(I) =p + is a local geometry indicator : N(I)(M) 0 : The point M is in a flat region. N(I)(M) 0 : The point M is on an edge. Following the behaviour of -function diffusions, we want an isotropic smoothing whenN(I) ' 0 and a tangent smoothing along the vector edge elsewhere (in the direction, coming from the Di Zenzo equations). Then, a natural extension of -functions diffusion for the vector case is : I t = 0( +) + I + 00( +) I + (I0 I) (4) where () is one of the -function used for the classic scalar case. Note that this PDE doesn’t come from a variational formulation, and diffusion coefficients can be chosen “by hand”, depending on the smoothing behaviour we desire. For instance, the following equation always diffuses the image, even on high gradients areas : I t = g (p +) I + I + (I0 I) where g (:) : R ! R is a decreasing function g (s) = exp s2 2 2 : (5) In this case, is a fixed parameter and represents the threshold between anisotropic and isotropic smoothing. The diffusion behaviour of these PDEs is : In homogeneous areas (g 1), the noise is removed efficiently due to a vector anisotropic diffusion which doesn’t favour any smoothing direction : I t ' I + I = I = 0BB I1 I2 : : : In 1CCA Along the edges (g ! 0), the diffusion is parallel to the vector contour : I t ' I = 0BB I1 I2 : : : In 1CCA There is noise elimination and vector contour conservation. ) The coupling is strongly used in order to analyse a local vector geometry of the image, and so perform a coherent smoothing process. Weighted data attachment term avoid the solution being too different from the initial image. The result at convergence is not over-smoothed. 3.4. Comparisons on a synthetic color image : We tested the described methods on a very noisy color synthetic image (fig.4). It shows the different behaviours of the diffusion equations. Pure isotropic PDE clears the noise very well, but also edges (fig.4c). Sapiro-Ringach PDE eq.(2) introduces some texture effects in flat regions (fig.4d) TVnm equation eq.(3) suffers of color blending (particularly near the edges). Our diffusion PDE eq.(4) clears the noise very well in homogeneous areas, while preserving color edges. a) Initial Image b) Noisy Image c) Isotropic diffusion d) Sapiro-Ringach eq.2 e) Blomgren TVnm (eq.3) f) Our proposed PDE Fig. 4. Comparison on a synthetic color image 4. REDUCING THE BLUR EFFECT Reducing the blurred edges is a part of the image restoration process. In this section, we propose to extend the scalar shock filters method [23] to the vector case, using the geometric view of vector fields. Then, we couple shock filters and vector diffusion in a single vector image restoration equation. 4.1. Shock filters in vector case Scalar shock filters allow to enhance blurred edges without any knowledge of the convolution mask. It consists in raising the edges in the gradient direction rI : (Osher and Rudin [23]) : I t = sign(I ) krIk which has the following effect on the image (Fig.5 represents a slice of the local image, in therI direction).